System for tracking user manipulations within an environment

ABSTRACT

Systems, methods and for analyzing game control input data are disclosed. A machine-readable medium having embodied thereon instructions for analyzing game control input data is also disclosed.

CLAIM OF PRIORITY

This application is a continuation in part (CIP) of U.S. patentapplication Ser. No. 10/207,677, entitled, “MAN-MACHINE INTERFACE USINGA DEFORMABLE DEVICE”, filed on Jul. 27, 2002; U.S. patent applicationSer. No. 10/650,409, entitled, “AUDIO INPUT SYSTEM”, filed on Aug. 27,2003; U.S. patent application Ser. No. 10/663,236, entitled “METHOD ANDAPPARATUS FOR ADJUSTING A VIEW OF A SCENE BEING DISPLAYED ACCORDING TOTRACKED HEAD MOTION”, filed on Sep. 15, 2003; U.S. patent applicationSer. No. 10/759,782, entitled “METHOD AND APPARATUS FOR LIGHT INPUTDEVICE”, filed on Jan. 16, 2004; U.S. patent application Ser. No.10/820,469, entitled “METHOD AND APPARATUS TO DETECT AND REMOVE AUDIODISTURBANCES”, filed on Apr. 7, 2004; and U.S. patent application Ser.No. 11/301,673, entitled “METHOD FOR USING RELATIVE HEAD AND HANDPOSITIONS TO ENABLE A POINTING INTERFACE VIA CAMERA TRACKING”, filed onDec. 12, 2005, U.S. patent application Ser. No. 11/381,721, to Xiao DongMao, entitled ULTRA SMALL MICROPHONE ARRAY, (Attorney DocketSCEA05062US00), filed on May 4, 2006, application Ser. No. 11/381,728,to Xiao Dong Mao, entitled ECHO AND NOISE CANCELLATION, (Attorney DocketSCEA05064US00), filed on May 4, 2006, U.S. patent application Ser. No.11/381,725, to Xiao Dong Mao, entitled “METHODS AND APPARATUS FORTARGETED SOUND DETECTION”, (Attorney Docket SCEA05072US00), filed on May4, 2006, U.S. patent application Ser. No. 11/381,727, to Xiao Dong Mao,entitled “NOISE REMOVAL FOR ELECTRONIC DEVICE WITH FAR FIELD MICROPHONEON CONSOLE”, (Attorney Docket SCEA05073US00), filed on May 4, 2006, U.S.patent application Ser. No. 11/381,724, to Xiao Dong Mao, entitled“METHODS AND APPARATUS FOR TARGETED SOUND DETECTION ANDCHARACTERIZATION”, (Attorney Docket SCEA05079US00), filed on May 4,2006, U.S. patent application Ser. No. 11/381,721, to Xiao Dong Mao,entitled “SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTERINTERACTIVE PROCESSING”, (Attorney Docket SCEA04005JUMBOUS), filed onMay 4, 2006; all of which are hereby incorporated by reference.

This application claims benefit of U.S. Provisional Patent ApplicationNo. 60/718,145, entitled “AUDIO, VIDEO, SIMULATION, AND USER INTERFACEPARADIGMS”, filed Sep. 15, 2005, which is hereby incorporated byreference.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to co-pending application Ser. No. ______,to Xiao Dong Mao, entitled “METHODS AND APPARATUSES FOR ADJUSTING ALISTENING AREA FOR CAPTURING SOUNDS”, (Attorney Docket SCEA-00300) filedon May 4, 2006, the entire disclosures of which are incorporated hereinby reference. This application is also related to co-pending applicationSer. No. ______ to Xiao Dong Mao, entitled “METHODS AND APPARATUSES FORCAPTURING AN AUDIO SIGNAL BASED ON VISUAL IMAGE”, (Attorney DocketSCEA-00400), filed on May 4, 2006, the entire disclosures of which areincorporated herein by reference. This application is also related toco-pending application Ser. No. ______, to Xiao Dong Mao, entitled“METHODS AND APPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON ALOCATION OF THE SIGNAL”, (Attorney Docket SCEA-00500), filed on May 4,2006, the entire disclosures of which are incorporated herein byreference. This application is also related to co-pending applicationSer. No. ______, to Richard Marks et al., entitled “SELECTIVE SOUNDSOURCE LISTENING IN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING”,(Attorney Docket SCEA04005US01-SONYP045), filed on May 4, 2006, theentire disclosures of which are incorporated herein by reference. Thisapplication is also related to co-pending application Ser. No. ______,to Richard Marks et al., entitled “Computer Image and Audio Processingof Intensity and Input Devices for Interfacing With A Computer Program”,(Attorney Docket SONYP052), filed on May 4, 2006, the entire disclosuresof which are incorporated herein by reference.

1. This application is also related to co-pending application Ser. No.______, entitled “MULTI-INPUT GAME CONTROL MIXER”, (Attorney DocketSCEAMXR1), filed on the same day as this application, the entiredisclosures of which are incorporated herein by reference.

3. This application is also related to co-pending application Ser. No.______, entitled “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONALINPUT CONTROL”, (Attorney Docket SCEA06INRT1), filed on the same day asthis application, the entire disclosures of which are incorporatedherein by reference.

4. This application is also related to co-pending application Ser. No.______, entitled “INERTIALLY TRACKABLE HAND-HELD CONTROLLER”, (AttorneyDocket SCEA06INRT2), filed on the same day as this application, theentire disclosures of which are incorporated herein by reference.

5. This application is also related to co-pending application Ser. No.______, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TOVISUAL TRACKING”, (Attorney Docket SONYP058A), filed on the same day asthis application, the entire disclosures of which are incorporatedherein by reference.

6. This application is also related to co-pending application Ser. No.______, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TOINERTIAL TRACKING”, (Attorney Docket SONYP058B), filed on the same dayas this application, the entire disclosures of which are incorporatedherein by reference.

7. This application is also related to co-pending application Ser. No.______, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TOACOUSTICAL TRACKING”, (Attorney Docket SONYP058C), filed on the same dayas this application, the entire disclosures of which are incorporatedherein by reference.

8. This application is also related to co-pending application Ser. No.______, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTS TOMULTI-CHANNEL MIXED INPUT”, (Attorney Docket SONYP058D), filed on thesame day as this application, the entire disclosures of which areincorporated herein by reference.

9. This application is also related to co-pending application Ser. No.______, entitled “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OFA GAME CONTROLLER BODY”, (Attorney Docket SCEA05082US00), filed on thesame day as this application, the entire disclosures of which areincorporated herein by reference.

10. This application is also related to co-pending application Ser. No.______, entitled “SCHEME FOR TRANSLATING MOVEMENTS OF A HAND-HELDCONTROLLER INTO INPUTS FOR A SYSTEM”, (Attorney Docket 86324), filed onthe same day as this application, the entire disclosures of which areincorporated herein by reference.

11. This application is also related to co-pending application Ser. No.______, entitled “DETECTABLE AND TRACKABLE HAND-HELD CONTROLLER”,(Attorney Docket 86325), filed on the same day as this application, theentire disclosures of which are incorporated herein by reference.

12. This application is also related to co-pending application Ser. No.______, entitled “METHOD FOR MAPPING MOVEMENTS OF A HAND-HELD CONTROLLERTO GAME COMMANDS”, (Attorney Docket 86326), filed on the same day asthis application, the entire disclosures of which are incorporatedherein by reference.

13. This application is also related to co-pending application Ser. No.______, entitled “CONTROLLER WITH INFRARED PORT ((DESIGN PATENT))”,(Attorney Docket SCEA06007US00), filed on the same day as thisapplication, the entire disclosures of which are incorporated herein byreference.

14. This application is also related to co-pending application Ser. No.______, entitled “CONTROLLER WITH TRACKING SENSORS ((DESIGN PATENT))”,(Attorney Docket SCEA06008US00), filed on the same day as thisapplication, the entire disclosures of which are incorporated herein byreference.

15. This application is also related to co-pending application Ser. No.______, entitled “DYNAMIC TARGET INTERFACE”, (Attorney DocketSCEA06009US00), filed on the same day as this application, the entiredisclosures of which are incorporated herein by reference.

16. This application is also related to co-pending application Ser. No.______, entitled “TRACKED CONTROLLER DEVICE ((DESIGN))”, (AttorneyDocket SCEA06010US00), filed on the same day as this application, theentire disclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to human-computer interfacingand specifically to processing multi-channel input for tracking the usermanipulation of one or more controllers.

BACKGROUND OF THE INVENTION

Computer entertainment systems typically include a hand-held controller,game controller, or other controller. A user or player uses thecontroller to send commands or other instructions to the entertainmentsystem to control a video game or other simulation being played. Forexample, the controller may be provided with a manipulator which isoperated by the user, such as a joy stick. The manipulated variable ofthe joy stick is converted from an analog value into a digital value,which is sent to the game machine main frame. The controller may also beprovided with buttons that can be operated by the user.

It is with respect to these and other background information factorsthat the present invention has evolved.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a pictorial diagram illustrating a video game system thatoperates in accordance with an embodiment of the present invention;

FIG. 2 is a perspective view of a controller made in accordance with anembodiment of the present invention;

FIG. 3 is a three-dimensional schematic diagram illustrating anaccelerometer that may be used in a controller according to anembodiment of the present invention;

FIG. 4 is a block diagram of a system for mixing various control inputsaccording to an embodiment of the present invention;

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

Various embodiments of the methods, apparatus, schemes and systemsdescribed herein provide for the detection, capture and tracking of themovements, motions and/or manipulations of the entire controller bodyitself by the user. The detected movements, motions and/or manipulationsof the entire controller body by the user may be used as additionalcommands to control various aspects of the game or other simulationbeing played.

Detecting and tracking a user's manipulations of a game controller bodymay be implemented in different ways. For example, an inertial sensor,such as an accelerometer or gyroscope, an image capture unit, such as adigital camera, can be used with the computer entertainment system todetect motions of the hand-held controller body and transfer them intoactions in a game. Examples of tracking a controller with an inertialsensor are described, e.g., in U.S. patent application Ser. No. ______,entitled “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUTCONTROL” (Attorney Docket Number SCEA06INRT1), which is incorporatedherein by reference. Examples of tracking a controller using imagecapture are described, e.g., in U.S. patent application Ser. No. ______,entitled “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAMECONTROLLER BODY” (Attorney Docket Number SCEA05082US00), which isincorporated herein by reference. In addition, the controller and/or theuser may also be tracked acoustically using a microphone array andappropriate signal processing. Examples of such acoustic tracking aredescribed in U.S. patent application Ser. No. 11/381,721, which isincorporated herein by reference.

Acoustic sensing, inertial sensing and image capture can be usedindividually or in any combination to detect many different types ofmotions of the controller, such as for example up and down movements,twisting movements, side to side movements, jerking movements, wand-likemotions, plunging motions, etc. Such motions may correspond to variouscommands such that the motions are transferred into actions in a game.Detecting and tracking the user's manipulations of a game controllerbody can be used to implement many different types of games,simulations, etc., that allow the user to, for example, engage in asword or lightsaber fight, use a wand to trace the shape of items,engage in many different types of sporting events, engage in on-screenfights or other encounters, etc. A game program may configured to trackthe motion of the controller and recognize certain pre-recorded gesturesfrom the tracked motion. Recognition of one or more of these gesturesmay trigger a change in the game state.

In embodiments of the present invention controller path informationobtained from these different sources may be mixed prior to analysis forgesture recognition. The tracking data from different sources (e.g.,acoustic, inertial and image capture) may be mixed in a way thatimproves the likelihood of recognition of a gesture.

Referring to FIG. 1, there is illustrated a system 100 that operates inaccordance with an embodiment of the present invention. As illustrated,a computer entertainment console 102 may be coupled to a television orother video display 104 to display the images of the video game or othersimulation thereon. The game or other simulation may be stored on a DVD,CD, flash memory, USB memory, or other memory media 106 that is insertedinto the console 102. A user or player 108 manipulates a a gamecontroller 110 to control the video game or other simulation. As seen inFIG. 2, the game controller 110 includes an inertial sensor 112 thatproduces signals in response to the position, motion, orientation orchange in orientation of the game controller 1 10. In addition to theinertial sensor, the game controller 110 may include conventionalcontrol input devices, e.g., joysticks 111, buttons 113, R1, L1, and thelike.

During operation, the user 108 physically moves the controller 110. Forexample, the controller 110 may be moved in any direction by the user108, such as up, down, to one side, to the other side, twisted, rolled,shaken, jerked, plunged, etc. These movements of the controller 110itself may be detected and captured by the camera 112 by way of trackingthrough analysis of signals from the inertial sensor 112 in a mannerdescribed below.

Referring again to FIG. 1, the system 100 may optionally include acamera or other video image capturing device 114, which may bepositioned so that the controller 110 is within the camera's field ofview 116. Analysis of images from the image capturing device 114 may beused in conjunction with analysis of data from the inertial sensor 112.As shown in FIG. 2, the controller 110 may optionally be equipped withlight sources such as light emitting diodes (LEDs) 202, 204, 206, 208 tofacilitate tracking by video analysis. These may be mounted to a body ofthe controller 110. As used herein, the term “body” meant to describethe part of the game controller 110 that one would hold by hand (or wearif it were a wearable game controller).

Analysis of such video images for the purpose of tracking the controller110 are described, e.g., in U.S. patent application Ser. No. ______,entitled “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAMECONTROLLER BODY” (Attorney Docket Number SCEA05082US00), to inventors______, which is incorporated herein by reference. The console 110 mayinclude an acoustic transducer, such as a microphone array 118. Thecontroller 110 may also include an acoustic signal generator 210 (e.g.,a speaker) to provide a source of sound to facilitate acoustic trackingof the controller 110 with the microphone array 118 and appropriateacoustic signal processing, e.g., as described in U.S. patentapplication Ser. No. 11/381,724, which is incorporated herein byreference.

In general, signals from the inertial sensor 112 are used to generateposition and orientation data for the controller 110. Such data may beused to calculate many physical aspects of the movement of thecontroller 110, such as for example its acceleration and velocity alongany axis, its tilt, pitch, yaw, roll, as well as any telemetry points ofthe controller 110. As used herein, telemetry generally refers to remotemeasurement and reporting of information of interest to a system or tothe system's designer or operator.

The ability to detect and track the movements of the controller 110makes it possible to determine whether any predefined movements of thecontroller 110 are performed. That is, certain movement patterns orgestures of the controller 110 may be predefined and used as inputcommands for the game or other simulation. For example, a plungingdownward gesture of the controller 110 may be defined as one command, atwisting gesture of the controller 110 may be defined as anothercommand, a shaking gesture of the controller 110 may be defined asanother command, and so on. In this way the manner in which the user 108physically moves the controller 110 is used as another input forcontrolling the game, which provides a more stimulating and entertainingexperience for the user.

By way of example and without limitation, the inertial sensor 112 may bean accelerometer. FIG. 3 depicts an example of an accelerometer 300 inthe form of a simple mass 302 elastically coupled at four points to aframe 304, e.g., by springs 306, 308, 310, 312. Pitch and roll axes(indicated by X and Y, respectively) lie in a plane that intersects theframe. A yaw axis Z is oriented perpendicular to the plane containingthe pitch axis X and the roll axis Y. The frame 304 may be mounted tothe controller 110 in any suitable fashion. As the frame 304 (and thejoystick controller 110) accelerates and/or rotates the mass 302 maydisplace relative to the frame 304 and the springs 306, 208, 310, 312may elongate or compress in a way that depends on the amount anddirection of translational and/or rotational acceleration and/or theangle of pitch and/or roll and/or yaw. The displacement and of the mass302 and/or compression or elongation of the springs 306, 308, 310, 312may be sensed, e.g., with appropriate sensors 314, 316, 318, 320 andconverted to signals that depend in known or determinable way on theamount acceleration of pitch and/or roll.

There are a number of different ways to track the position of the massand/or or the forces exerted on it, including resistive strain gaugematerial, photonic sensors, magnetic sensors, hall-effect devices,piezoelectric devices, capacitive sensors, and the like. Embodiments ofthe invention may include any number and type or combination of types ofsensors. By way of example, and without limitation, the sensors 314,316, 318, 320 may be gap closing electrodes placed above the mass 302. Acapacitance between the mass and each electrode changes as the positionof the mass changes relative to each electrode. Each electrode may beconnected to a circuit that produce a signal related to the capacitance(and therefore to the proximity) of the mass 302 relative to theelectrode. In addition, the springs 306, 308, 310, 312 may includeresistive strain gauge sensors that produce signals that are related tothe compression or elongation of the springs.

In some embodiments, the frame 304 may be gimbal mounted to thecontroller 110 so that the accelerometer 300 maintains a fixedorientation with respect to the pitch and/or roll and/or yaw axes. Insuch a manner, the controller axes X, Y, Z may be directly mapped tocorresponding axes in real space without having to take into account atilting of the controller axes with respect to the real space coordinateaxes.

As discussed above, data from inertial, image capture and acousticsources may be analyzed to generate a path that tracks the positionand/or orientation of the controller 110. As shown in the block diagramof FIG. 4 a system 400 according to an embodiment of the invention mayinclude an inertial analyzer 402, an image analyzer 404 and an acousticanalyzer 406. Each of these analyzers receives signals from a sensedenvironment 401. The analyzers 402, 406, 408 may be implemented inhardware, in software (or firmware) or some combination of two or moreof these. Each of the analyzers produces tracking information relatedthe position and/or orientation of an object of interest. By way ofexample, the object of interest may be the controller 110 referred toabove. The image analyzer 404 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. ______ (Attorney Docket Number 86321SCEA05082US00).The inertial analyzer 402 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. ______, entitled “SYSTEM, METHOD, AND APPARATUS FORTHREE-DIMENSIONAL INPUT CONTROL” (Attorney Docket Number SCEA06INRT1).The acoustic analyzer 406 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. 11/381,724.

The analyzers 402, 404 and 406 may be regarded as being associated withdifferent channels of inputs of position and/or orientation information.The Mixer 408 may accept multiple input channels and such channels maycontain sample data characterizing the sensed environment 401, typicallyfrom the perspective of the channel. The position and/or orientationinformation generated by the inertial analyzer 402, image analyzer 404and acoustic analyzer 406 can be coupled into the input of a mixer 408.The Mixer 408 and analyzers 402, 404, 406 may be queried by a gamesoftware 410 and may be configured to interrupt game software inresponse to events. Events may include gesture recognition events,gearing changes, configuration changes, setting noise levels, settingsampling rate, changing mapping chains, etc, examples of which arediscussed below. The mixer 408 may operate in connection with and toform fields below and with respect to methods described herein.

As discussed above, signals from different input channels, e.g.,inertial sensors, video images and/or acoustic sensors may be analyzedby the inertial analyzer 402, image analyzer 404 and acoustic analyzer406, respectively, to determine the motion and/or orientation of thecontroller 110 during play of a video game according to an inventivemethod. Such a method may be implemented as a series of processorexecutable program code instructions stored in a processor readablemedium and executed on a digital processor. For example, as depicted inFIG. 5A, the video game system 100 may include on the console 102 havingthe inertial analyzer 402, image analyzer 404 and acoustic analyzer 406implemented either in hardware or software. By way of example, theanalyzers 402, 404, 406 may be implemented as software instructionsrunning on a suitable processor unit 502. By way of example, theprocessor unit 502 may be a digital processor, e.g., a microprocessor ofa type commonly used in video game consoles. A portion of theinstructions may be stored in a memory 506. Alternatively, the inertialanalyzer 402, image analyzer 404 and acoustic analyzer 406 may beimplemented in hardware, e.g., as an application specific integratedcircuit (ASIC). Such analyzer hardware may be located on the controller110 or on the console 102 or may be remotely located elsewhere. Inhardware implementations, the analyzers 402, 404, 406 may beprogrammable in response to external signals e.g., from the processor502 or some other remotely located source, e.g., connected by USB cable,wireless connection, or over a network.

The inertial analyzer 402 may include or implement instructions thatanalyze the signals generated by the inertial sensor 112 and utilizeinformation regarding position and/or orientation of the controller 110.Similarly, the image analyzer 404 may implement instructions thatanalyze images captured by the image capture unit 114. In addition, theacoustic analyzer may implement instructions that analyze imagescaptured by the microphone array 118. As shown in the flow diagram 510of FIG. 5B these signals and/or images may be received by the analyzers402, 404, 406 as indicated at block 512. The signals and/or images maybe analyzed by the analyzers 402, 404, 406 to determine inertialtracking information 403, image tracking information 405 and acoustictracking information 407 regarding the position and/or orientation ofthe controller 110 as indicated at block 514. The tracking information403, 405, 407 may be related to one or more degrees of freedom. It ispreferred that six degrees of freedom are tracked to characterize themanipulation of the controller 110 or other tracked object. Such degreesof freedom may relate to the controller tilt, yaw, roll and position,velocity or acceleration along the x, y and z-axis.

As indicated at block 516, the mixer 408 mixes the inertial information403, image information 405 and acoustic information 407 to generaterefined position and/or orientation information 409. By way of example,the mixer 408 may apply different weights the inertial, image andacoustic tracking information 403, 405, 407 based on game orenvironmental conditions and the take a weighted average. In addition,the mixer 408 may include its own mixer analyzer 412 that analyzes thecombined position/orientation information and generates its ownresulting “mixer” information that involves combinations of theinformation generated by the other analyzers.

In an embodiment of the present invention the mixer 408 may assign adistribution value to the tracking information 403, 405, 407 from theanalyzers 402, 404, 406. As noted above, certain sets of input controldata may be averaged. In the present embodiment, however, the inputcontrol data is assigned a value prior to its being averaged whereby theinput control data from some analyzers is of more analytical importancethan from others.

The mixer 408 may take on a number of functionalities in the context ofthe present system including observation, correction, stabilization,derivation, combination, routing, mixing, reporting, buffering,interrupting other processes and analysis. Such may be performed withrespect to the tracking information 403, 405, 407 received from one ormore of the analyzers 402, 404, 406. While each of the analyzers 402,404, 406 may receive and/or derive certain tracking information, themixer 408 may be implemented to optimize the use of the receivedtracking information 403, 405, 407 and generate refined trackinginformation 409.

The analyzers 402, 404, 406 and mixer 408 are preferably configured toprovide tracking information similar output formats. Trackinginformation parameters from any analyzer element 402, 404, 406 can bemapped to a single parameter in an analyzer. Alternatively, the mixer408 may form tracking information for any of the analyzers 402, 404, 406by processing one or more tracking information parameters from one ormore of analyzers 402, 404, 406. The mixer may combine two or moreelements of tracking information of the same parameter type taken fromthe analyzers 402, 404, 406 and/or perform functions across multipleparameters of tracking information generated by the analyzers to createa synthetic set of output having the benefit of being generated frommultiple channels of input.

The refined tracking information 409 may be utilized during play of avideo game with the system 100 as indicated at block 518. In certainembodiments, the position and/or orientation information may be used inrelation to gestures made by the user 108 during game play. In someembodiments the mixer 408 may operate in conjunction with the gesturerecognizer 505 to associate at least one action in a game environmentwith one or more user actions from the user (e.g., manipulation of thecontroller in space).

As indicated in the flow diagram 520 of FIG. 5C, a path of thecontroller 110 may be tracked using the position and/or orientationinformation as indicated at block 522. By way of example, and withoutlimitation, the path may include a set of points representing a positionof the center of mass of the controller with respect to some system ofcoordinates. Each position point may be represented by one or morecoordinates, e.g., X, Y and Z coordinates in a Cartesian coordinatesystem. A time may be associated with each point on the path so thatboth the shape of the path and the progress of the controller along thepath may be monitored. In addition, each point in the set may haveassociated with it data representing an orientation of the controller,e.g., one or more angles of rotation of the controller about its centerof mass. Furthermore, each point on the path may have associated with itvalues of velocity and acceleration of the center of mass of thecontroller and rates of angular rotation and angular acceleration of thecontroller about its center of mass.

As indicated at block 524, the tracked path may be compared to one ormore stored paths corresponding to known and/or pre-recorded gestures508 that are relevant to the context of the video game being played.Recognizer 505 may be configured to recognize a user or process audioauthenticated gestures, etc. For example, a user may be identified bythe recognizer 505 through a gesture and that a gesture may be specificto a user. Such a specific gestures may recorded and included among thepre-recorded gestures 508 stored in memory 506. The recordation processmay optionally store audio generated during recordation of a gesture.The sensed environment is sampled into a multi-channel analyzer andprocessed. The processor may reference gesture models to determine andauthenticate and/or identify a user or objects based on voice oracoustic patterns and to a high degree of accuracy and performance.

As indicated in FIG. 5A, data 508 representing the gestures may bestored in the memory 506. Examples of gestures include, but are notlimited to throwing an object such as a ball, swinging an object such asa bat or golf club, pumping hand pump, opening or closing a door orwindow, turning steering wheel or other vehicle control, martial artsmoves such as punches, sanding movements, wax on wax off, paint thehouse, shakes, rattles, rolls, football pitches, turning knob movements,3D MOUSE movements, scrolling movements, movements with known profiles,any recordable movement, movements along any vector back and forth i.e.pump the tire but at some arbitrary orientation in space, movementsalong a path, movements having precise stop and start times, any timebased user manipulation that can be recorded, tracked and repeatedwithin the noise floor, splines, and the like. Each of these gesturesmay be pre-recorded from path data and stored as a time-based model.Comparison of the path and stored gestures may start with an assumptionof a steady state if the path deviates from a steady state the path canbe compared to the stored gestures by a process of elimination. If atblock 526 there is no match, the analyzer 504 may continue tracking thepath of the controller 110 at block 522. If there is a sufficient matchbetween the path (or a portion thereof) and a stored gesture the stateof the game may be changed as indicated at 528. Changes of state of thegame may include, but are not limited to interrupts, sending controlsignals, changing variables, etc.

Here is one example of this can occur. Upon determining that thecontroller 110 has left a steady state the path an analyzer 402, 404,406, or 412 tracks movement of the controller 110. As long as the pathof the controller 110 complies with a path defined in the stored gesturemodels 508, those gestures are possible “hits”. If the path of thecontroller 110 deviates (within the noise tolerance setting) from anygesture model 508, that gesture model is removed from the hit list. Eachgesture reference model includes a time-base in which the gesture isrecorded. The analyzer 402, 404, 406, or 412 compares the controllerpath data to the stored gestures 508 at the appropriate time index.Occurrence of a steady state condition resets the clock. When deviatingfrom steady state (i.e. when movements are tracked outside of the noisethreshold) the hit list is populated with all potential gesture models.The clock is started and movements of the controller are comparedagainst the hit list. Again, the comparison is a walk through time. Ifany gesture in the hit list reaches the end of the gesture then it is ahit.

In certain embodiments, the mixer 408 and/or individual analyzers 402,404, 406, 412 may inform a game program when certain events occur.Examples of such events include the following:

INTERRUPT ZERO-ACCELERATION POINT REACHED (X AND/OR Y AN/OR Z AXIS) Incertain game situations the analyzer 504 may notify or interrupt routinewithin the game program when acceleration of the controller changes atthe inflection points. For example, the user 108 may use the controller110 to control a game avatar representing a quarterback in a footballsimulation game. The analyzer 504 may track the controller (representingthe football) via a path generated from signals from the inertial sensor112. A particular change in acceleration of the controller 110 maysignal release of the football. At this point, the analyzer may triggeranother routine within the program (e.g., a physics simulation package)to simulate the trajectory of the football based on the position, and/orvelocity and/or orientation of the controller at the point of release.

Interrupt New Gesture Recognized

In addition, the analyzer 502 may be configured by one or more inputs.Examples of such inputs include, but are not limited to:

SET NOISE LEVEL (X, Y or Z AXIS) The noise level may be a referencetolerance used when analyzing jitter of the user's hands in the game.

SET SAMPLING RATE. As used herein, the sampling rate may refer to howoften the analyzer 502 samples the signals from the inertial sensor. Thesampling rate may be set to oversample or average the signal.

SET GEARING. As used herein gearing generally refers to the ratio ofcontroller movements to movements occurring within the game. Examples ofsuch “gearing” in the context of control of a video game may be found inU.S. patent application Ser. No. ______, filed ______, (Attorney DocketNo.:______), which is incorporated herein by reference.

SET MAPPING CHAIN. As used herein, a mapping chain refers to a map ofgesture models. The gesture model maps can be made for a specific inputChannel (e.g., for path data generated from inertial sensor signalsonly) or for a hybrid Channel formed in a mixer unit. Three inputChannels may be served by two or more different Analyzers that aresimilar to the inertial analyzer 504. Specifically, these may include:the inertial analyzer 504 as described herein, a video analyzer asdescribed e.g., in U.S. patent application Ser. No. ______, to inventors______, entitled SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OFA GAME CONTROLLER BODY (Attorney Docket SCEA05082US00), which isincorporated herein by reference, and an Acoustic Analyzer, e.g., asdescribed in U.S. patent application Ser. No. 11/381,721, which isincorporated herein by reference. The Analyzers can be configured with amapping chain. Mapping chains can be swapped out by the game duringgameplay as can settings to the Analyzer and to the Mixer.

Referring to again to FIG. 5A, block 502, those of skill in the art willrecognize that there are numerous ways to generate signals from theinertial sensor 112. A few examples, among others have been describedabove with respect to FIGS. 3A-3E. Referring to block 504, there arenumerous ways to analyze the sensor signals generated in block 502 toobtain information relating to the position and/or orientation of thecontroller 110. By way of example and without limitation the trackinginformation may include, but is not limited to information regarding thefollowing parameters individually or in any combination:

CONTROLLER ORIENTATION. Orientation of the controller 110 may beexpressed in terms of pitch, roll or yaw angle with respect to somereference orientation, e.g., in radians). Rates of change of controllerorientation (e.g., angular velocities or angular accelerations) may alsobe included in the position and/or orientation information. Where theinertial sensor 112 includes a gyroscopic sensor controller orientationinformation may be obtained directly in the form of one or more outputvalues that are proportional to angles of pitch, roll or yaw.

CONTROLLER POSITION (e.g., Cartesian coordinates X, Y, Z of thecontroller 110 in some frame of reference)

CONTROLLER X-AXIS VELOCITY

CONTROLLER Y-AXIS VELOCITY

CONTROLLER Z-AXIS VELOCITY

CONTROLLER X-AXIS ACCELERATION

CONTROLLER Y-AXIS ACCELERATION

CONTROLLER Z-AXIS ACCELERATION

It is noted that with respect to position, velocity and acceleration theposition and/or orientation information may be expressed in terms ofcoordinate systems other than Cartesian. For example, cylindrical orspherical coordinates may be used for position, velocity andacceleration. Acceleration information with respect to the X, Y and Zaxes may be obtained directly from an accelerometer type sensor, e.g.,as described above with respect to FIGS. 3A-3E. The X, Y and Zaccelerations may be integrated with respect to time from some initialinstant to determine changes in X, Y and Z velocities. These velocitiesmay be computed by adding the velocity changes to known values of theX-, Y-, and Z-velocities at the initial instant in time. The X, Y and Zvelocities may be integrated with respect to time to determine X-, Y-,and Z-displacements of the controller. The X-, Y-, and Z-positions maybe determined by adding the displacements to known X-, Y-, and Z-,positions at the initial instant.

STEADY STATE Y/N—This particular information indicates whether thecontroller is in a steady state, which may be defined as any position,which may be subject to change too. In a preferred embodiment the steadystate position may be one wherein the controller is held in a more orless level orientation at a height roughly even with a user's waist.

TIME SINCE LAST STEADY STATE generally refers to data related to howlong a period of time has passed since a steady state (as referencedabove) was last detected. That determination of time may, as previouslynoted, be calculated in real-time, processor cycles, or samplingperiods. The Time Since Last Steady State data time may be importantwith regard to resetting tracking of a controller with regard to aninitial point to ensure accuracy of character or object mapping in agame environment. This data may also be important with regard todetermining available actions/gestures that might be subsequentlyexecuted in a game environment (both exclusively and inclusively).

LAST GESTURE RECOGNIZED generally refers to the last gesture recognizedeither by the gesture recognizer 505 (which may be implemented inhardware or software. The identification of a last gesture recognizedmay be important with respect to the fact that a previous gesture may berelated to the possible gestures that may be subsequently recognized orsome other action that takes place in the game environment.

Time Last Gesture Recognized

The above outputs can be sampled at any time by a game program orsoftware.

In an embodiment of the present invention the mixer 408 may assign adistribution value to the tracking information 403, 405, 407 from theanalyzers 402, 404, 406. As noted above, certain sets of input controldata may be averaged. In the present embodiment, however, the inputcontrol data is assigned a value prior to its being averaged whereby theinput control data from some analyzers is of more analytical importancethan from others.

For example, the mixer 408 may require tracking information related toacceleration and steady state. The mixer 408 would then receive thetracking information 403, 405, 407 as described above. The trackinginformation may include parameters relating to acceleration and steadystate, e.g., as described above. Prior to averaging the datarepresenting this information, the mixer 408 may assign distributionvalues to tracking information data set 403, 405, 407. For example, thex- and y-acceleration parameters from the inertial analyzer 402 may beweighted at a value of 90%. The x- and y-acceleration data from theimage analyzer 406, however, may be weighted at only 10%. The acousticanalyzer tracking information 407 as it pertains to accelerationparameters may be weighted at zero percent, that is, the data has novalue.

Similarly, the Z-axis tracking information parameters from the inertialanalyzer 402 may be weighted at 10% whereas the image analyzer Z-axistracking information may be weighted at 90%. The acoustic analyzertracking information 407 may, again, be weighted at 0% value but steadystate tracking information from the acoustic analyzer 406 may beweighted at 100% with the remaining analyzer tracking information beweighted at 0%.

After the appropriate distribution weight has been assigned, the inputcontrol data may be averaged in conjunction with that weight to arriveat a weighted average input control data set that is subsequentlyanalyzed by the gesture recognizer 505 and associated with a particularaction in the game environment. The values associated may be pre-definedby the mixer 408 or by a particular game title. The values may also bethe result of the mixer 408 identifying a particular quality of datacoming from the various analyzers and thus making a dynamic adjustmentas is further discussed below. The adjustment may also be the result ofbuilding a historical knowledge base of when particular data is ofparticular value in a particular environment and/or in response to theparticularities of a given game title.

The mixer 408 may be configured to operate dynamically during game play.For example, as the mixer 408 receives various input control data, itmay recognize that certain data is consistently outside an acceptablerange or quality of data or reflects corrupt data that may be indicativeof a processing error at the related input device.

Additionally, certain conditions in a real-world environment mightchange. For example, natural light in the user's at-home gameenvironment might be increasing as the morning turns to afternooncausing problems with image data capture. Further, a neighborhood orhousehold might become noisier as the day goes on causing problems withaudio data capture. Likewise, if a user has been playing for severalhours, their reflexes may become less sharp thus causing problems withinterpretation of inertial data.

In these instances, or in any other instance wherein the quality of aparticular form of input control data is in question, the mixer 408 maydynamically reassign distribution weight to a particular set of datacoming from a particular device such that more or less importance isgiven to particular input control data as described above. Similarly,the game environment may change over the course of the game wherein theneeds of a particular game change thus requiring a reassignment of valueor need for particular input control data.

Similarly, the mixer 408 may recognize that certain data being passed onto the gesture recognizer 505 is being processed incorrectly, slowly, ornot at all based on processing errors or feedback data that may begenerated by the gesture recognizer 505. In response to this feedback orin recognition of these processing difficulties (e.g., while the imageanalysis data is within an acceptable range, errors result when anassociation is made by the gesture recognizer 505), the mixer 408 mayadjust what input control data it seeks from what analyzer and when, ifat all. The mixer 408 may further require certain analysis andprocessing of input control data by the proper analyzer before it ispassed to the mixer 408, which may re-process the data (e.g., averagethe data) such that a further layer of assurance is made that the datapassed to the gesture recognizer 505 will be processed effectively andappropriately.

In some embodiments, the mixer 408 may recognize that certain data iscorrupt, ineffective, or outside a particular variable and may call uponparticular input control data or variable related to that data such thatit may replace incorrect data or properly analyze and calculate certaindata with respect to the necessary variables.

According to embodiments of the present invention, a video game systemand method of the type described above may be implemented as depicted inFIG. 6. A video game system 600 may include a processor 601 and a memory602 (e.g., RAM, DRAM, ROM, and the like). In addition, the video gamesystem 600 may have multiple processors 601 if parallel processing is tobe implemented. The memory 602 includes data and game program code 604,which may include portions that are configured as described above.Specifically, the memory 602 may include inertial signal data 606 whichmay include stored controller path information as described above. Thememory 602 may also contain stored gesture data 608, e.g., datarepresenting one or more gestures relevant to the game program 604.Coded instructions executed on the processor 602 may implement amulti-input mixer 605, which may be configured and function as describedabove.

The system 600 may also include well-known support functions 610, suchas input/output (I/O) elements 611, power supplies (P/S) 612, a clock(CLK) 613 and cache 614. The apparatus 600 may optionally include a massstorage device 615 such as a disk drive, CD-ROM drive, tape drive, orthe like to store programs and/or data. The controller may alsooptionally include a display unit 616 and user interface unit 618 tofacilitate interaction between the controller 600 and a user. Thedisplay unit 616 may be in the form of a cathode ray tube (CRT) or flatpanel screen that displays text, numerals, graphical symbols or images.The user interface 618 may include a keyboard, mouse, joystick, lightpen or other device. In addition, the user interface 618 may include amicrophone, video camera or other signal transducing device to providefor direct capture of a signal to be analyzed. The processor 601, memory602 and other components of the system 600 may exchange signals (e.g.,code instructions and data) with each other via a system bus 620 asshown in FIG. 6.

A microphone array 622 may be coupled to the system 600 through the I/Ofunctions 611. The microphone array may include between about 2 andabout 8 microphones, preferably about 4 microphones with neighboringmicrophones separated by a distance of less than about 4 centimeters,preferably between about 1 centimeter and about 2 centimeters.Preferably, the microphones in the array 622 are omni-directionalmicrophones. An optional image capture unit 623 (e.g., a digital camera)may be coupled to the apparatus 600 through the I/O functions 611. Oneor more pointing actuators 625 that are mechanically coupled to thecamera may exchange signals with the processor 601 via the I/O functions611.

As used herein, the term I/O generally refers to any program, operationor device that transfers data to or from the system 600 and to or from aperipheral device. Every data transfer may be regarded as an output fromone device and an input into another. Peripheral devices includeinput-only devices, such as keyboards and mouses, output-only devices,such as printers as well as devices such as a writable CD-ROM that canact as both an input and an output device. The term “peripheral device”includes external devices, such as a mouse, keyboard, printer, monitor,microphone, game controller, camera, external Zip drive or scanner aswell as internal devices, such as a CD-ROM drive, CD-R drive or internalmodem or other peripheral such as a flash memory reader/writer, harddrive.

In certain embodiments of the invention, the apparatus 600 may be avideo game unit, which may include a controller 630 coupled to theprocessor via the I/O functions 611 either through wires (e.g., a USBcable) or wirelessly. The controller 630 may have analog joystickcontrols 631 and conventional buttons 633 that provide control signalscommonly used during playing of video games. Such video games may beimplemented as processor readable data and/or instructions from theprogram 604 which may be stored in the memory 602 or other processorreadable medium such as one associated with the mass storage device 615.

The joystick controls 631 may generally be configured so that moving acontrol stick left or right signals movement along the X axis, andmoving it forward (up) or back (down) signals movement along the Y axis.In joysticks that are configured for three-dimensional movement,twisting the stick left (counter-clockwise) or right (clockwise) maysignal movement along the Z axis. These three axis—X Y and Z—are oftenreferred to as roll, pitch, and yaw, respectively, particularly inrelation to an aircraft.

In addition to conventional features, the controller 630 may include oneor more inertial sensors 632, which may provide position and/ororientation information to the processor 601 via an inertial signal.Orientation information may include angular information such as a tilt,roll or yaw of the controller 630. By way of example, the inertialsensors 632 may include any number and/or combination of accelerometers,gyroscopes or tilt sensors. In a preferred embodiment, the inertialsensors 632 include tilt sensors adapted to sense orientation of thejoystick controller with respect to tilt and roll axes, a firstaccelerometer adapted to sense acceleration along a yaw axis and asecond accelerometer adapted to sense angular acceleration with respectto the yaw axis. An accelerometer may be implemented, e.g., as a MEMSdevice including a mass mounted by one or more springs with sensors forsensing displacement of the mass relative to one or more directions.Signals from the sensors that are dependent on the displacement of themass may be used to determine an acceleration of the joystick controller630. Such techniques may be implemented by instructions from the gameprogram 604 which may be stored in the memory 602 and executed by theprocessor 601.

By way of example an accelerometer suitable as the inertial sensor 632may be a simple mass elastically coupled at three or four points to aframe, e.g., by springs. Pitch and roll axes lie in a plane thatintersects the frame, which is mounted to the joystick controller 630.As the frame (and the joystick controller 630) rotates about pitch androll axes the mass will displace under the influence of gravity and thesprings will elongate or compress in a way that depends on the angle ofpitch and/or roll. The displacement and of the mass can be sensed andconverted to a signal that is dependent on the amount of pitch and/orroll. Angular acceleration about the yaw axis or linear accelerationalong the yaw axis may also produce characteristic patterns ofcompression and/or elongation of the springs or motion of the mass thatcan be sensed and converted to signals that are dependent on the amountof angular or linear acceleration. Such an accelerometer device canmeasure tilt, roll angular acceleration about the yaw axis and linearacceleration along the yaw axis by tracking movement of the mass orcompression and expansion forces of the springs. There are a number ofdifferent ways to track the position of the mass and/or or the forcesexerted on it, including resistive strain gauge material, photonicsensors, magnetic sensors, hall-effect devices, piezoelectric devices,capacitive sensors, and the like.

In addition, the joystick controller 630 may include one or more lightsources 634, such as light emitting diodes (LEDs). The light sources 634may be used to distinguish one controller from the other. For exampleone or more LEDs can accomplish this by flashing or holding an LEDpattern code. By way of example, 5 LEDs can be provided on the joystickcontroller 630 in a linear or two-dimensional pattern. Although a lineararray of LEDs is preferred, the LEDs may alternatively, be arranged in arectangular pattern or an arcuate pattern to facilitate determination ofan image plane of the LED array when analyzing an image of the LEDpattern obtained by the image capture unit 623. Furthermore, the LEDpattern codes may also be used to determine the positioning of thejoystick controller 630 during game play. For instance, the LEDs canassist in identifying tilt, yaw and roll of the controllers. Thisdetection pattern can assist in providing a better user/feel in games,such as aircraft flying games, etc. The image capture unit 623 maycapture images containing the joystick controller 630 and light sources634. Analysis of such images can determine the location and/ororientation of the joystick controller. Such analysis may be implementedby program code instructions 604 stored in the memory 602 and executedby the processor 601. To facilitate capture of images of the lightsources 634 by the image capture unit 623, the light sources 634 may beplaced on two or more different sides of the joystick controller 630,e.g., on the front and on the back (as shown in phantom). Such placementallows the image capture unit 623 to obtain images of the light sources634 for different orientations of the joystick controller 630 dependingon how the joystick controller 630 is held by a user.

In addition the light sources 634 may provide telemetry signals to theprocessor 601, e.g., in pulse code, amplitude modulation or frequencymodulation format. Such telemetry signals may indicate which joystickbuttons are being pressed and/or how hard such buttons are beingpressed. Telemetry signals may be encoded into the optical signal, e.g.,by pulse coding, pulse width modulation, frequency modulation or lightintensity (amplitude) modulation. The processor 601 may decode thetelemetry signal from the optical signal and execute a game command inresponse to the decoded telemetry signal. Telemetry signals may bedecoded from analysis of images of the joystick controller 630 obtainedby the image capture unit 623. Alternatively, the apparatus 601 mayinclude a separate optical sensor dedicated to receiving telemetrysignals from the lights sources 634. The use of LEDs in conjunction withdetermining an intensity amount in interfacing with a computer programis described, e.g., in U.S. patent application Ser. No. ______, toRichard L. Marks et al., entitled “USE OF COMPUTER IMAGE AND AUDIOPROCESSING IN DETERMINING AN INTENSITY AMOUNT WHEN INTERFACING WITH ACOMPUTER PROGRAM” (Attorney Docket No. SONYP052), filed May 4, 2006,which is incorporated herein by reference in its entirety. In addition,analysis of images containing the light sources 634 may be used for bothtelemetry and determining the position and/or orientation of thejoystick controller 630. Such techniques may be implemented byinstructions of the program 604 which may be stored in the memory 602and executed by the processor 601.

The processor 601 may use the inertial signals from the inertial sensor632 in conjunction with optical signals from light sources 634 detectedby the image capture unit 623 and/or sound source location andcharacterization information from acoustic signals detected by themicrophone array 622 to deduce information on the location and/ororientation of the controller 630 and/or its user. For example,“acoustic radar” sound source location and characterization may be usedin conjunction with the microphone array 622 to track a moving voicewhile motion of the joystick controller is independently tracked(through the inertial sensor 632 and or light sources 634). In acousticradar select a pre-calibrated listening zone is selected at runtime andsounds originating from sources outside the pre-calibrated listeningzone are filtered out. The pre-calibrated listening zones may include alistening zone that corresponds to a volume of focus or field of view ofthe image capture unit 623. Examples of acoustic radar are described indetail in U.S. patent application Ser. No. 11/381,724, to Xiadong Maoentitled “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION ANDCHARACTERIZATION”, filed May 4, 2006, which is incorporated herein byreference. Any number of different combinations of different modes ofproviding control signals to the processor 601 may be used inconjunction with embodiments of the present invention. Such techniquesmay be implemented by program code instructions 604 which may be storedin the memory 602 and executed by the processor 601 and may optionallyinclude one or more instructions that direct the one or more processorsto select a pre-calibrated listening zone at runtime and filter outsounds originating from sources outside the pre-calibrated listeningzone. The pre-calibrated listening zones may include a listening zonethat corresponds to a volume of focus or field of view of the imagecapture unit 623.

The program 604 may optionally include one or more instructions thatdirect the one or more processors to produce a discrete time domaininput signal x_(m)(t) from microphones M₀ . . . M_(M), of the microphonearray 622, determine a listening sector, and use the listening sector ina semi-blind source separation to select the finite impulse responsefilter coefficients to separate out different sound sources from inputsignal x_(m)(t). The program 604 may also include instructions to applyone or more fractional delays to selected input signals x_(m)(t) otherthan an input signal x₀(t) from a reference microphone M₀. Eachfractional delay may be selected to optimize a signal to noise ratio ofa discrete time domain output signal y(t) from the microphone array. Thefractional delays may be selected to such that a signal from thereference microphone M₀ is first in time relative to signals from theother microphone(s) of the array. The program 604 may also includeinstructions to introduce a fractional time delay Δ into an outputsignal y(t) of the microphone array so that:y(t+Δ)=x(t+Δ)*b₀+x(t−1+Δ)*b₁+x(t−2+Δ)*b₂+ . . . +x(t−N+Δ)b_(N), where Δis between zero and ±1. Examples of such techniques are described indetail in U.S. patent application Ser. No. 11/381,729, to Xiadong Mao,entitled “ULTRA SMALL MICROPHONE ARRAY” filed May 4, 2006, the entiredisclosures of which are incorporated by reference.

The program 604 may include one or more instructions which, whenexecuted, cause the system 600 to select a pre-calibrated listeningsector that contains a source of sound. Such instructions may cause theapparatus to determine whether a source of sound lies within an initialsector or on a particular side of the initial sector. If the source ofsound does not lie within the default sector, the instructions may, whenexecuted, select a different sector on the particular side of thedefault sector. The different sector may be characterized by anattenuation of the input signals that is closest to an optimum value.These instructions may, when executed, calculate an attenuation of inputsignals from the microphone array 622 and the attenuation to an optimumvalue. The instructions may, when executed, cause the apparatus 600 todetermine a value of an attenuation of the input signals for one or moresectors and select a sector for which the attenuation is closest to anoptimum value. Examples of such a technique are described, e.g., in U.S.patent application Ser. No. 11/381,725, to Xiadong Mao, entitled“METHODS AND APPARATUS FOR TARGETED SOUND DETECTION” filed May 4, 2006,the disclosures of which are incorporated herein by reference.

Signals from the inertial sensor 632 may provide part of a trackinginformation input and signals generated from the image capture unit 623from tracking the one or more light sources 634 may provide another partof the tracking information input. By way of example, and withoutlimitation, such “mixed mode” signals may be used in a football typevideo game in which a Quarterback pitches the ball to the right after ahead fake head movement to the left. Specifically, a game player holdingthe controller 630 may turn his head to the left and make a sound whilemaking a pitch movement swinging the controller out to the right like itwas the football. The microphone array 620 in conjunction with “acousticradar” program code can track the user's voice. The image capture unit623 can track the motion of the user's head or track other commands thatdo not require sound or use of the controller. The sensor 632 may trackthe motion of the joystick controller (representing the football). Theimage capture unit 623 may also track the light sources 634 on thecontroller 630. The user may release of the “ball” upon reaching acertain amount and/or direction of acceleration of the joystickcontroller 630 or upon a key command triggered by pressing a button onthe controller 630.

In certain embodiments of the present invention, an inertial signal,e.g., from an accelerometer or gyroscope may be used to determine alocation of the controller 630. Specifically, an acceleration signalfrom an accelerometer may be integrated once with respect to time todetermine a change in velocity and the velocity may be integrated withrespect to time to determine a change in position. If values of theinitial position and velocity at some time are known then the absoluteposition may be determined using these values and the changes invelocity and position. Although position determination using an inertialsensor may be made more quickly than using the image capture unit 623and light sources 634 the inertial sensor 632 may be subject to a typeof error known as “drift” in which errors that accumulate over time canlead to a discrepancy D between the position of the joystick 630calculated from the inertial signal (shown in phantom) and the actualposition of the joystick controller 630. Embodiments of the presentinvention allow a number of ways to deal with such errors.

For example, the drift may be cancelled out manually by re-setting theinitial position of the controller 630 to be equal to the currentcalculated position. A user may use one or more of the buttons on thecontroller 630 to trigger a command to re-set the initial position.Alternatively, image-based drift may be implemented by re-setting thecurrent position to a position determined from an image obtained fromthe image capture unit 623 as a reference. Such image-based driftcompensation may be implemented manually, e.g., when the user triggersone or more of the buttons on the joystick controller 630.Alternatively, image-based drift compensation may be implementedautomatically, e.g., at regular intervals of time or in response to gameplay. Such techniques may be implemented by program code instructions604 which may be stored in the memory 602 and executed by the processor601.

In certain embodiments it may be desirable to compensate for spuriousdata in the inertial sensor signal. For example the signal from theinertial sensor 632 may be oversampled and a sliding average may becomputed from the oversampled signal to remove spurious data from theinertial sensor signal. In some situations it may be desirable tooversample the signal and reject a high and/or low value from somesubset of data points and compute the sliding average from the remainingdata points. Furthermore, other data sampling and manipulationtechniques may be used to adjust the signal from the inertial sensor toremove or reduce the significance of spurious data. The choice oftechnique may depend on the nature of the signal, computations to beperformed with the signal, the nature of game play or some combinationof two or more of these. Such techniques may be implemented byinstructions of the program 604 which may be stored in the memory 602and executed by the processor 601.

The processor 601 may perform analysis of inertial signal data 606 asdescribed above in response to the data 606 and program codeinstructions of a program 604 stored and retrieved by the memory 602 andexecuted by the processor module 601. Code portions of the program 604may conform to any one of a number of different programming languagessuch as Assembly, C++, JAVA or a number of other languages. Theprocessor module 601 forms a general-purpose computer that becomes aspecific purpose computer when executing programs such as the programcode 604. Although the program code 604 is described herein as beingimplemented in software and executed upon a general purpose computer,those skilled in the art will realize that the method of task managementcould alternatively be implemented using hardware such as an applicationspecific integrated circuit (ASIC) or other hardware circuitry. As such,it should be understood that embodiments of the invention can beimplemented, in whole or in part, in software, hardware or somecombination of both.

In one embodiment, among others, the program code 604 may include a setof processor readable instructions that implement a method havingfeatures in common with the method 510 of FIG. 5B and the method 520 ofFIG. 5C or some combination of two or more of these. The program code604 may generally include one or more instructions that direct the oneor more processors to analyze signals from the inertial sensor 632 togenerate position and/or orientation information and utilize theinformation during play of a video game.

The program code 604 may optionally include processor executableinstructions including one or more instructions which, when executedcause the image capture unit 623 to monitor a field of view in front ofthe image capture unit 623, identify one or more of the light sources634 within the field of view, detect a change in light emitted from thelight source(s) 634; and in response to detecting the change, triggeringan input command to the processor 601. The use of LEDs in conjunctionwith an image capture device to trigger actions in a game controller isdescribed e.g., in U.S. patent application Ser. No. 10/759,782 toRichard L. Marks, filed Jan. 16, 2004 and entitled: METHOD AND APPARATUSFOR LIGHT INPUT DEVICE, which is incorporated herein by reference in itsentirety.

The program code 604 may optionally include processor executableinstructions including one or more instructions which, when executed,use signals from the inertial sensor and signals generated from theimage capture unit from tracking the one or more light sources as inputsto a game system, e.g., as described above. The program code 604 mayoptionally include processor executable instructions including one ormore instructions which, when executed compensate for drift in theinertial sensor 632.

Although embodiments of the present invention are described in terms ofexamples related to a video game controller 630 games, embodiments ofthe invention, including the system 600 may be used on any usermanipulated body, molded object, knob, structure, etc, with inertialsensing capability and inertial sensor signal transmission capability,wireless or otherwise.

By way of example, embodiments of the present invention may beimplemented on parallel processing systems. Such parallel processingsystems typically include two or more processor elements that areconfigured to execute parts of a program in parallel using separateprocessors. By way of example, and without limitation, FIG. 7illustrates a type of cell processor 700 according to an embodiment ofthe present invention. The cell processor 700 may be used as theprocessor 601 of FIG. 6 or the processor 502 of FIG. 5A. In the exampledepicted in FIG. 7, the cell processor 700 includes a main memory 702,power processor element (PPE) 704, and a number of synergistic processorelements (SPEs) 706. In the example depicted in FIG. 7, the cellprocessor 700 includes a single PPE 704 and eight SPE 706. In such aconfiguration, seven of the SPE 706 may be used for parallel processingand one may be reserved as a back-up in case one of the other sevenfails. A cell processor may alternatively include multiple groups ofPPEs (PPE groups) and multiple groups of SPEs (SPE groups). In such acase, hardware resources can be shared between units within a group.However, the SPEs and PPEs must appear to software as independentelements. As such, embodiments of the present invention are not limitedto use with the configuration shown in FIG. 7.

The main memory 702 typically includes both general-purpose andnonvolatile storage, as well as special-purpose hardware registers orarrays used for functions such as system configuration, data-transfersynchronization, memory-mapped I/O, and I/O subsystems. In embodimentsof the present invention, a video game program 703 may be resident inmain memory 702. The video program 703 may include inertial, image andacoustic analyzers and a mixer configured as described with respect toFIGS. 4, 5A, 5B or 5C above or some combination of these. The program703 may run on the PPE. The program 703 may be divided up into multiplesignal processing tasks that can be executed on the SPEs and/or PPE.

By way of example, the PPE 704 may be a 64-bit PowerPC Processor Unit(PPU) with associated caches L1 and L2. The PPE 704 is a general-purposeprocessing unit, which can access system management resources (such asthe memory-protection tables, for example). Hardware resources may bemapped explicitly to a real address space as seen by the PPE. Therefore,the PPE can address any of these resources directly by using anappropriate effective address value. A primary function of the PPE 704is the management and allocation of tasks for the SPEs 706 in the cellprocessor 700.

Although only a single PPE is shown in FIG. 7, some cell processorimplementations, such as cell broadband engine architecture (CBEA), thecell processor 700 may have multiple PPEs organized into PPE groups, ofwhich there may be more than one. These PPE groups may share access tothe main memory 702. Furthermore the cell processor 700 may include twoor more groups SPEs. The SPE groups may also share access to the mainmemory 702. Such configurations are within the scope of the presentinvention.

Each SPE 706 is includes a synergistic processor unit (SPU) and its ownlocal storage area LS. The local storage LS may include one or moreseparate areas of memory storage, each one associated with a specificSPU. Each SPU may be configured to only execute instructions (includingdata load and data store operations) from within its own associatedlocal storage domain. In such a configuration, data transfers betweenthe local storage LS and elsewhere in the system 700 may be performed byissuing direct memory access (DMA) commands from the memory flowcontroller (MFC) to transfer data to or from the local storage domain(of the individual SPE). The SPUs are less complex computational unitsthan the PPE 704 in that they do not perform any system managementfunctions. The SPU generally have a single instruction, multiple data(SIMD) capability and typically process data and initiate any requireddata transfers (subject to access properties set up by the PPE) in orderto perform their allocated tasks. The purpose of the SPU is to enableapplications that require a higher computational unit density and caneffectively use the provided instruction set. A significant number ofSPEs in a system managed by the PPE 704 allow for cost-effectiveprocessing over a wide range of applications.

Each SPE 706 may include a dedicated memory flow controller (MFC) thatincludes an associated memory management unit that can hold and processmemory-protection and access-permission information. The MFC providesthe primary method for data transfer, protection, and synchronizationbetween main storage of the cell processor and the local storage of anSPE. An MFC command describes the transfer to be performed. Commands fortransferring data are sometimes referred to as MFC direct memory access(DMA) commands (or MFC DMA commands).

Each MFC may support multiple DMA transfers at the same time and canmaintain and process multiple MFC commands. Each MFC DMA data transfercommand request may involve both a local storage address (LSA) and aneffective address (EA). The local storage address may directly addressonly the local storage area of its associated SPE. The effective addressmay have a more general application, e.g., it may be able to referencemain storage, including all the SPE local storage areas, if they arealiased into the real address space.

To facilitate communication between the SPEs 706 and/or between the SPEs706 and the PPE 704, the SPEs 706 and PPE 704 may include signalnotification registers that are tied to signaling events. The PPE 704and SPEs 706 may be coupled by a star topology in which the PPE 704 actsas a router to transmit messages to the SPEs 706. Alternatively, eachSPE 706 and the PPE 704 may have a one-way signal notification registerreferred to as a mailbox. The mailbox can be used by an SPE 706 to hostoperating system (OS) synchronization.

The cell processor 700 may include an input/output (I/O) function 708through which the cell processor 700 may interface with peripheraldevices, such as a microphone array 712 and optional image capture unit713 and a game controller 730. The game controller unit may include aninertial sensor 732, and light sources 734. In addition an ElementInterconnect Bus 710 may connect the various components listed above.Each SPE and the PPE can access the bus 710 through a bus interfaceunits BIU. The cell processor 700 may also includes two controllerstypically found in a processor: a Memory Interface Controller MIC thatcontrols the flow of data between the bus 710 and the main memory 702,and a Bus Interface Controller BIC, which controls the flow of databetween the I/O 708 and the bus 710. Although the requirements for theMIC, BIC, BIUs and bus 710 may vary widely for differentimplementations, those of skill in the art will be familiar theirfunctions and circuits for implementing them.

The cell processor 700 may also include an internal interrupt controllerIIC. The IIC component manages the priority of the interrupts presentedto the PPE. The IIC allows interrupts from the other components the cellprocessor 700 to be handled without using a main system interruptcontroller. The IIC may be regarded as a second level controller. Themain system interrupt controller may handle interrupts originatingexternal to the cell processor.

In embodiments of the present invention, certain computations, such asthe fractional delays described above, may be performed in parallelusing the PPE 704 and/or one or more of the SPE 706. Each fractionaldelay calculation may be run as one or more separate tasks thatdifferent SPE 706 may take as they become available.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A” or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

1. A system for tracking user manipulations within an environment,comprising: a first analyzer operable to receive first input channelinformation via a first input channel, the first input channelinformation being representative of the user manipulations within theenvironment, the first analyzer further being operable to analyze thefirst input channel information to generate first output information forregistering the user manipulations by a game program during execution ofthe game program.
 2. The system as claimed in claim 1, furthercomprising a second analyzer operable to receive second input channelinformation from a second input channel, the second input channelinformation being representative of the user manipulations within theenvironment, the second analyzer further being operable to analyze thesecond channel information to generate second output information forregistering the user manipulations by a game program during execution ofthe game program; and a programmable mixer operable to selectinformation regarding a first aspect of the user manipulations from thefirst output information and to select information regarding a secondaspect of the user manipulations from the second output information andto generate mixed output information for supply to the game programbased on the selected first output information and the selected secondoutput information.
 3. The system as claimed in claim 2, furthercomprising a third analyzer operable to receive third input channelinformation from a third input channel, the third input channelinformation being representative of the user manipulations within theenvironment, the third analyzer further being operable to analyze thethird channel information to generate third output information forregistering the user manipulations by a game program during execution ofthe game program, wherein the programmable mixer is operable to selectinformation regarding a third aspect of the user manipulations from thethird output information and to generate the mixed output informationfor supply to the game program based on each of the selected firstoutput information, selected second output information and selectedthird output information.
 4. The system as claimed in claim 2, whereinthe first input channel information includes information obtainableusing a first type of sensor and the second input channel informationincludes information obtainable using a second type of sensor differentfrom the first type of sensor.
 5. The system as claimed in claim 4,wherein the first input channel information includes informationobtainable through manipulation of a hand-held controller.
 6. The systemas claimed in claim 4, wherein the first input channel informationincludes information obtainable by tracking movement of at least aportion of a user's body.
 7. The system as claimed in claim 4, whereinthe first analyzer includes an inertial analyzer and the first inputchannel information includes inertial information obtainable by aninertial sensor.
 8. The system as claimed in claim 6, wherein theinertial sensor includes at least one of an accelerometer or agyroscope.
 9. The system as claimed in claim 4, wherein the secondanalyzer includes a video analyzer and the second input channelinformation includes information indicative of a position of auser-movable object within an image obtainable from an image capturedevice.
 10. The system as claimed in claim 9, wherein the image capturedevice includes a camera.
 11. The system as claimed in claim 8, whereinthe second analyzer includes an acoustic analyzer and the second inputchannel information includes information obtainable by transducing auser-manipulable source of sound with an acoustic pick-up device. 12.The system as claimed in claim 11, wherein the acoustic pick-up deviceincludes a microphone.
 13. The system as claimed in claim 11, whereinthe second input channel information includes information obtainable bytransducing a voice within the environment.
 14. The system as claimed inclaim 11, wherein the second input channel information includesinformation indicative of a position of a voice within the environment.15. The system as claimed in claim 11, wherein the second input channelinformation includes information indicative of a position of a beaconwithin the environment.
 16. The system as claimed in claim 11, whereinthe second input channel information includes information indicative ofa position of a user-movable beacon within the environment.
 17. Thesystem as claimed in any one of claims 1 through 10 or claim 13, whereinthe first analyzer includes an inertial analyzer and the first inputchannel information includes inertial information obtainable by aninertial sensor, the second analyzer includes a video analyzer and thesecond input channel information includes information indicative of aposition of a user-movable object within an image obtainable from animage capture device, and the third analyzer includes an acousticanalyzer and the third input channel information includes informationobtainable by transducing a user-manipulable source of sound with anacoustic pick-up device.
 18. The system as claimed in claim 17, whereinthe second input channel information includes information indicative ofa position of the user-manipulable source of sound within theenvironment.
 19. The system as claimed in claim 1, 2, 17 or 18 whereinat least some of the first input channel information, second inputchannel information or third input channel information is representativeof an acceleration of a user-manipulable object within the environment.20. The system as claimed in claim 1, 2, 17, 18 or 19 wherein at leastsome of the first input channel information, second input channelinformation or third input channel information is representative of avelocity of a user-manipulable object within the environment.
 21. Thesystem as claimed in claim 1, 2, 17, 18 or 19 wherein at least some ofthe first input channel information, second input channel information orthird input channel information is representative of an orientation of auser-manipulable object within the environment.
 22. The system asclaimed in claim 21, wherein the orientation of the user-manipulableobject includes information indicative of at least one of pitch, yaw androll.
 23. The system as claimed in claim 21, wherein the orientation ofthe user-manipulable object includes information indicative of pitch,yaw and roll.
 24. The system as claimed in claim 1, 2 or in any ofclaims 17 through 23, wherein at least some of the first input channelinformation, second input channel information or third input channelinformation is representative of the position of the user-manipulableobject within a coordinate space defined by at least two orthogonalaxes.
 25. The system as claimed in claim 24, wherein the at least someinformation is representative of the position of the user-manipulableobject within a coordinate space defined by at least three orthogonalaxes.
 26. The system as claimed in claim 2 or in any one of claims 17through 25, wherein the programmable mixer is operable to select theinformation regarding the first aspect of the user manipulationsexclusively from the first output information and to select theinformation regarding the second aspect of the user manipulationsexclusively from the second output information.
 27. The system asclaimed in claim 26, wherein the programmable mixer is operable toselect the information regarding the first aspect of the usermanipulations partially from the first output information and partiallyfrom the second output information.
 28. The system as claimed in claim 2or in any one of claims 17 through 25, wherein the programmable mixer isoperable to generate the output information regarding the first aspectof the user manipulations by averaging the first output informationregarding the first aspect with the second output information regardingthe first aspect.
 29. The system as claimed in claim 28, wherein thefirst output information and the second output information are assigneddifferent weights and the mixer is operable to perform the averaging asa weighted average in accordance with the different weights.
 30. Thesystem as claimed in claims 1 or 2 wherein at least one of: the firstanalyzer is operable to store a model of a gesture and to analyze thefirst input channel information over an interval of time to detect agesture when user manipulations match the model, and to generate asignal upon detecting the gesture.
 31. The system as claimed in claims1, 2 or in any one of claims 17 through 25, wherein at least one of theanalyzer or the mixer is operable to identify a user based on analysisof the input channel information.
 32. The system as claimed in claims 1,2 or in any one of claims 17 through 25 or 31, wherein at least one ofthe analyzer or the mixer is operable to authenticate a user based onanalysis of the input channel information.
 33. The system as claimed inclaims 1, 2 or in any one of claims 17 through 25 or 31, wherein atleast one of the analyzer or the mixer is operable to authenticate theuser in response to identifying the user.
 34. The system as claimed inclaims 1, 2 or in any one of claims 17 through 25, 31 or 32, wherein atleast one of the analyzer or the mixer is operable to analyze the inputchannel information to discriminate between the user manipulations of afirst user and a second user.
 35. The system as claimed in claims 1, 2or in any one of claims 17 through 25, 31 or 32, wherein at least one ofthe analyzer or the mixer is operable to analyze the input channelinformation to generate the output information to include firstsubchannel output information for registering the user manipulations ofa first user to the game program and second subchannel outputinformation separable from the first output information for registeringthe user manipulations of a second user to the game program.
 36. Thesystem as claimed in any one of claims 30 through 35, wherein the inputchannel information includes image input channel information obtainablefrom an image capture device.
 37. The system as claimed in any one ofclaims 30 through 35, wherein the input channel information includesinertial sensing information obtainable from an inertial sensor selectedfrom at least one of an accelerometer or a gyroscope.
 38. The system asclaimed in any one of claims 30 through 35, wherein the input channelinformation includes acoustic input channel information obtainable froman acoustic transducer.